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FOREWORD 


The  Instructional  Quality  Inventory  (IQI)  was  developed  in  support  of 
Navy  Decision  Coordinating  Paper,  Education  and  Training  Development 
(NDCP-Z0108-PN),  under  subproject  P.30A,  Adaptive  Experimental  Approach 
to  Instructional  Design,  and  the  sponsorship  of  the  Director  of  Naval 
Education  and  Training  (OP-99).  The  overall  objective  of  the  subproject 
is  to  develop  an  empirically-based  instructional  design  support  system 
to  aid  developers  in  deciding  on  instructional  alternatives  based  on 
cost/benefits  and  specified  resource  limitations.  The  purpose  of  the 
IQI,  which  was  originally  called  the  Instructional  Strategy  Diagnostic 
Profile  (ISDP),  is  to  provide  quality  control  and/or  evaluation  procedures 
for  instructional  development. 

A  number  of  reports  have  been  published  on  the  IQI/ISDP.  The  first  pro¬ 
vided  an  interim  training  manual  for  the  ISDP  (NPRDC  Special  Report  77-14), 
and  the  next  three  addressed  its  empirical  and  workshop  evaluations  (NPRDC 
Technical  Reports  77-25  and  77-43  and  Special  Report  78-17).  As  a  result 
of  these  evaluations,  the  ISDP  was  extensively  revised  and  retitled  as 
the  IQI.  These  revisions  were  included  in  NPRDC  Technical  Note  78-5, 
which  provided  an  interim  training  manual  for  the  IQI. 

This  report  is  the  fourth  in  a  series  of  four  on  IQI  procedures.  It  provides 
a  Job  Performance  Aid  for  the  IQI  process,  including  a  list  of  all  the  IQI 
procedures.  The  other  three  reports  are: 

1.  The  Instructional  Quality  Inventory:  Volume  I,  Introduction  and 
Overview  (NPRDC  Special  Report  79-3). 

2.  An  IQI  User's  Manual,  which  will  include  a  complete  description  of 

all  IQI  procedures,  and  provide  examples  of  their  use  (to  be  published 
in  early  1979). 

3.  An  IQI  Training  Workbook,  which  will  provide  additional  examples,  and 
practice  on  the  IQI  procedures  (to  be  published  in  early  1979). 

When  these  four  reports  have  been  published,  previous  training  manuals  (NPRDC 
Special  Report  77-14  and  Technical  Note  78-5)  will  have  been  superseded. 

The  IQI  is  intended  for  use  by  the  Chief  of  Naval  Education  and  Training; 
the  Chief  of  Naval  Technical  Training;  the  Chief  of  Naval  Education  and 
Training  Support  (specifically,  the  Instructional  Program  Development  Centers); 
the  Commander  Training  Command,  Atlantic;  the  Commander  Training  Conmand, 
Pacific;  and  all  other  Navy  activities  concerned  with  the  development,  revision, 
or  acquisition  of  instructional  programs.  Prospective  users  of  the  IQI  are 
invited  to  contact  this  comnand  for  assistance  in  implementation. 


DONALD  F.  PARKER 
Commanding  Officer 
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OBJECTIVE  ADEQUACY 

STEP  1:  ENTER  the  COURSE  TITLE  and  OBJECTIVE  NUMBER  at  the  top  of  the  form. 

STEP  2:  Determine  whether  or  not  the  OBJECTIVE  is  CORRECTLY  STATED. 

2a:  Are  the  CONDITIONS  under  which  student  performance  Is  expected  specified? 

ENVIRONMENT:  PHYSICAL  (weather,  time  of  day,  lighting,  etc.) 

SOCIAL  {isolation,  individual,  team,  audience,  etc.) 
PSYCHOLOGICAL  (fatigue,  stress,  relaxed,  etc.) 

INFORMATION:  GIVEN  INFORMATION  (scenario,  formula,  values,  etc.) 

CUES  (signals  for  starting  or  stopping) 

SPECIAL  INSTRUCTIONS 

RESOURCES:  JOB  AIDS  (cards,  charts,  graphs,  checklists,  etc.) 

EQUIPMENT,  TOOLS 
TECHNICAL  MANUALS 

2b:  Are  the  STANDARDS  which  the  student  performance  must  meet  specified? 

PERFORMANCE:  COMPLETENESS  (how  much  of  the  task  must  be  performed) 
ACCURACY  (how  well  must  each  task  be  performed) 

TIME  LIMIT  (how  much  time  is  allowed) 

RATE  (how  fast  must  task  be  done) 

PRODUCT:  COMPLETENESS  (what  must  finished  product  contain) 

QUALITY  (what  objective  standard  must  product  meet) 
JUDGEMENT  (what  subjective  opinions  must  product  satisfy) 
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2c:  Is  the  ACTION  the  student  must  perform  specified? 

Is  an  action  verb  used  to  specify  what  the  student  must  do? 
Is  only  one  action  stated  in  the  objective? 

STEP  3:  Determine  whether  or  not  the  OBJECTIVE  is  CLASSIFIABLE?  Does  the  OBJECTIVE  fit 
in  one  and  only  one  cell  of  the  table  below? 


FACT 

RECALL  OR 
RECOGNIZE 
NAMES, 

PARTS; 

DATES; 

PLACES;  ETC. 

REMEMBER  -  recall  OR  I 

RECOGNIZE  FACTS;  CON¬ 
CEPT  DEFINITIONS;  STEPS 
OF  PROCEDURES  OR  RULES; 

STATEMENTS  OF  PRINCIPLE 

USE-UNAIDED  -  tasks  which  require 

CLASSIFYING,  PERFORMING  A  PROCEDURE, 
USING  A  RULE,  EXPLAINING  OR  PREDICTING 
WITH  NO  AIDS  EXCEPT  MEMORY. 


USE-AIDED  -  SAME  AS  USE-UNAIDED; 
EXCEPT  JOB  AIDS  ARE  AVAILABLE. 


CONCEPT 

REMEMBER 
CHARACTERIS¬ 
TICS,  OR 
CLASSIFY  OB¬ 
JECTS,  EVENTS 
OR  IDEAS  AC¬ 
CORDING  TO 
CHARACTERISTICS 


PROCEDURE 

SEQUENCE  OF 
STEPS  REMEM¬ 
BERED  OR  USED 
IN  A  SINGLE 
SITUATION  OR 
ON  A  SINGLE 
PIECE  OF 
EQUIPMENT 


RULE 

REMEMBER  OR 
USE  A  SEQUENCE 
OF  STEPS  WHICH 
APPLY  ACROSS 
SITUATIONS 
OR  ACROSS 
EQUIPMENTS 


PRINCIPLE 

REMEMBER,  OR 
INTERPRET  OR 
PREDICT,  WHY 
OR  HOW  THINGS 
HAPPEN,  OR 
CAUSE-EFFECT 
RELATIONSHIPS 


(Continued  on  other  aide) 
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TEST  CONSISTENCY 

CARD  2 

STEP  1:  ENTER  the  COURSE  TITLE  and  OBJECTIVE  NUMBER  at  the  top  of  the  form. 

STEP  2:  ENTER  the  TEST  ITEM  NUMBERS  for  the  items  associated  with  this  objective  on  the  form. 

STEP  3:  DETERMINE  whether  or  not  the  CONDITIONS  in  each  item,  or  the  CONDITIONS  under  which  the 
items  are  administered,  match  the  conditions  in  the  objective. 

STEP  4:  DETERMINE  whether  or  not  the  STANDARDS  in  each  item,  or  the  STANDARDS  for  scoring  each 
item,  match  the  standards  in  the  objective. 

STEP  5:  DETERMINE  whether  or  not  the  ACTIONS  in  each  item  match  the  action  of  the  objective. 

STEP  5a:  Determine  the  TASK  LEVEL  and  CONTENT  TYPE  of  each  test  item. 

STEP  5b:  Determine  whether  these  match  the  TASK  LEVEL  and  CONTENT  TYPE  of  the  objective. 

STEP  6:  DETERMINE  whether  or  not  each  Item  is  typical  of  the  job  to  be  performed  after  training, 
or  is  a  necessary  qualification  for  later  training. 

STEP  7:  DETERMINE  whether  or  not  the  FORMAT  of  each  item  is  APPROPRIATE  for  the  TASK  LEVEL  and 
CONTENT  TYPE.  Use  the  table  below: 


CONTENT  TYPE 


TASK  LEVEL 

FACT 

CONCEPT 

PROCEDURE 

RULE 

PRINCIPLE 

REMEMBER 

for  RECOGNITION: 

matching 
true-false 
multiple  choice 
for  RECALL: 
short  answer 
fill-in 
listing 

short  answer 

fill-in 

listing 

short  answer 

fill-in 

listing 

short  answer 

fill-in 

listing 

short  answer 

fill-in 

listing 

USE-UNAIDED 

performance 
matching 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 

USE-AIDED 

performance 
ma tching 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 

performance 
true-false 
multiple  choice 
short  answer 
fill-in 
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TEST  ADEQUACY 

CARD  3 

STEP  1:  DETERMINE  whether  or  not  each  item  is  CLEAR.  Instructions  for  completing  the  item  muet  specify 
what  response  the  student  is  expected  to  make. 

STEP  2:  DETERMINE  whether  or  not  each  item  is  UNAMBIGUOUS.  Each  item  must  have  one  and  only  one  correct 

response,  and  the  item  must  be  interpretable  in  only  one  way.  That  is,  items  must  not  be  confusing. 

STEP  3:  DETERMINE  whether  or  not  each  item  is  WELL  CONSTRUCTED.  Different  criteria  apply  to  different 
item  formats : 

TRUE-FALSE  items:  An  item  should  include  only  one  statement  to  be 

judged  true  or  false. 

Negative  statements  should  be  avoided. 

Don't  use  words  like  "never,"  "always,"  etc. 

Item  statements  should  be  short. 


MULTIPLE  CHOICE  items: 


MATCHING  items: 


FILL-IN  items: 


SHORT  ANSWER  items: 


LISTING  items: 


PERFORMANCE  items: 


All  alternatives  should  be  plausible. 

Negatives  in  the  item  stem  should  be  highlighted. 

Repetitive  phrases  should  be  placed  in  the  stem, 
not  in  the  alternatives. 

Alternatives  like  "all  of  the  above,  "  "A  and  B  only,  " 
should  be  avoided. 

Instructions  should  explain  the  contents  of  each  column, 
and  explain  the  basis  for  matching. 

Instructions  should  specify  how  many  times  each  answer 
may  be  used. 

The  choice  column  should  include  extra  answers  unless 
answers  may  be  used  more  than  once. 

The  blank  should  be  at  or  near  the  end  of  the  sentence. 

One  and  only  one  phrase  should  correctly  complete  the  item. 

Multiple  blanks  should  be  avoided. 

Blanks  should  require  key  words. 

The  required  answer  should  be  kept  short. 

The  directions  to  the  student  should  specify  how  the 
item  will  be  scored. 

The  scoring  key  should  identify  allowable  synonyms  or 
alternatives. 

The  directions  should  specify  the  number  of  things  to 
be  listed  (if  appropriate  for  the  objective,  and  if 
the  number  of  things  is  not  a  hint) . 

The  directions  should  specify  whether  or  not  order  is 
important.  If  so,  the  scoring  key  should  score 
sequence  separately. 

The  scoring  key  should  identify  allowable  synonyms  or 
alternatives,  and  should  specify  different  weights 
if  appropriate. 

The  directions  should  clearly  explain  what  the  student 
is  to  do  and  how  the  item  will  be  scored. 

The  scoring  key  must  specify  all  criteria  the  performance 
must  meet,  such  as  completeness,  accuracy,  quality, 
time  limit,  rate,  etc.  If  steps  in  the  performance 
are  scored,  a  checklist  should  be  provided. 


STEP  4:  DETERMINE  whether  or  not  each  item  is  FREE  of  HINTS.  An  item  should  not  give  away  the  answer 
to  itself  or  to  any  other  item  on  the  test.  The  grarmar  of  multiple-choice  and  fill-in  items 
should  not  give  hints  to  answers. 

STEP  5:  DETERMINE  whether  or  not  the  items  allow  for  COMMON  ERRORS  to  be  made. 

STEP  6:  For  USE-level  objectives,  DETERMINE  whether  or  not  there  are  ENOUGH  ITEMS  to  test  the 
objective  adequately,  and  to  reflect  the  range  of  performance  required  on  the  job. 


PRESENTATION  CONSISTENCY 
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step  1 

STEP  2 
STEP  3 


ENTER  the  COURSE  TITLE  and  OBJECTIVE  NUMBER  at  the  top  of  the  form. 
LOCATE  the  section  of  the  presentation  related  to  this  objective. 


DETERMINE  if  the  required  PRESENTATION  COMPONENTS  are  present  for  the  task  level 
of  the  objective.  (Use  the  table  below). 


TASK  LEVEL 

STATEMENT 

PRACTICE 

REMEMBERING 

EXAMPLES 

» 

REMEMBER 

required 

required 

not 

required 

not 

required 

USE-UNAIDED 

required,  un 
associated 
objective 
recent lu 

less  the 
REMEMBER 
as  taught 

required 

required 

USE-AIDED 

the  aid 

rep  laces 
statement 

not 

required 

required 
with,  aid 

required 
with  aid 

STEP  4:  DETERMINE  whether  or  not  each  required  PRESENTATION  COMPONENT  is  COMPLETE  for  the  content 
type  of  the  objective  (Use  the  table  below).  Examples  and  practice  items  must  also  match 
the  task  level  of  the  objective. 


PRESENTATION 

COMPONENT _ CONTENT  TYPE  OF  THE  OBJECTIVE 


FACT 

CONCEPT 

PROCEDURE 

RULE 

PRINCIPLE 

STATEMENT 

complete  fact 
presented 

all  critica  1 
characteristics 
and  their 
combination 
are  given 

all  steps  are 
given  in  the 
correct  order 

all  steps  and 
branching 
decisions  are 
given  in  the 
correct  order 

all  causes, 
effects,  and 
relationships 
are  given 

PRACTICE 

REMEMBERING 

recall  or  recog¬ 
nition  required 

recall  of 
concept  def¬ 
inition  required 

recall  of  all 
steps  in  correct 
order  required 

recall  of  all 
steps  and  branch 
decisions  in 
correct  order 
required 

recall  of  all 
causes,  effects, 
relationships 
required 

For  all  content  types:  Practice  Remembering  items  must 
They  must  be  the  same  format  as 
All  practice  items  must  include 

be  the  same  as  the 
the  test  item, 
feedback. 

test  item. 

EXAMPLES 

not 

applicable 

examples  show 
all  critical 
characteristics 
required  for 
classification, 
non-examples  show 
absence  of 
critical 
characteristics 

application  of 
the  procedure 
must  be  shown 
and  steps  must 
be  shown  in  the 
correct  order 

application  of 
each  step  or 
branching 
decision  must 
be  shown  in  the 
correct  order 

interpretation 
or  prediction 
based  on  causes, 
effects,  and 
relationships 
must  be  shown 

PRACTICE 

USING 

not 

applicable 

classification 
of  both  examples 
and  non-examples 
is  required 

all  steps  must 
be  performed 
in  the  correct 
order 

all  steps  and 
branching 
decisions  must 
be  performed  in 
the  correct  order 

explanation  or 
prediction 
based  on  the 
principle  is 
requi red 

For  all  content  types:  Practice  Using  Items  must  reflect  what  is  to  be  done  on  the 
job  or  in  later  training. 

The  task/content  level,  conditions,  and  standards  must  match 
the  test  item  and  objective. 

The  practice  Item  format  must  be  the  same  as  the  test  item  format. 
All  practice  items  must  Include  feedback. 


For  CONCEPTS,  RULES,  Some  practice  items  should  be  different  than  either  the  test 
and  PRINCIPLES:  items  or  the  examples.  (Common  error  items  might  be  the  same.) 


.****.  riu  «r- 


STEP  1: 


PRESENTATION  ADEQUACY 
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DETERMINE  whether  or  not  each  required  PRESENTATION  COMPONENT  meets  the  general 
ADEOOACY  criteria  below. 

SEPARATED:  Statements,  Examples,  or  Practice  must  be  SEPARATED  from  the 

rest  of  the  Instruction.  There  are  different  ways  components 
may  be  separated: 


a.  Set  off  the  component  with  a  box. 

b.  Use  a  different  color  or  type  face,  or  underline. 

a.  Place  on  a  separate  page,  or  in  a  special  place  on  the  page, 
d.  For  audio,  movies,  or  lectures,  pause  before  introducing  the 
component. 


IDENTIFIED: 


Statements,  Examples,  and  Practice  must  be  IDENTIFIED  so  the 
student  knows  what  they  are,  and  can  locate  them.  Labels  can 
be  used  to  identify  different  components: 

Definition  of  .. .  Example  Practice 

Procedure  for  . . .  Demonstration  Test  Yourself 

Key  Point: 


CLEARLY  STATED:  Statements,  Examples,  and  Practice  must  be  CLEARLY  STATED  so  the 
student  can  understand  them.  The  following  criteria  should  be 
used: 


a.  The  READING  LEVEL  must  be  appropriate  for  the  students. 

b.  The  presentation  should  not  be  confusing,  vague,  or  too  Wordy. 

c.  All  essential  information  should  be  present;  the  student 
should  not  be  referred  to  other  places  to  obtain  information. 

d.  All  presentations  should  be  PERFORMANCE-ORIENTED,  not 
topic-oriented. 


STEP  2:  DETERMINE  whether  or  not  each  required  PRESENTATION  COMPONENT  meets  the  specific  ADEQUACY 
criteria  below: 

STATEMENTS:  HELP:  In  addition  to  the  statement,  the  instruction  should  include  something 

to  help  the  student  better  understand  and  remember  the  statement. 

Methods  of  providing  help  include: 

a.  Giving  a  MNEMONIC  (memory  trick)  or  other  memory  aid. 

b.  Representing  the  statement  with  pictures,  symbols,  flowcharts,  etc. 

c.  Explaining  how  the  statement  relates  to  something  the  student 
already  knows,  how  it  fits  in  the  course,  why  it  is  important. 

d.  Giving  more  explanation  about  what  the  statement  means. 

CLEARLY  STATED:  In  addition  to  the  criteria  above,  statements  for  different  content  types 
must  meet  the  criteria  below: 


(Continued  on  other  side) 


CONCEPTS:  Give  a  decision  rule  or  search  strategy  for  classification. 

PROCEDURES:  Each  step  should  have  only  one  action. 

RULES:  If  a  formula  is  used,  symbols  must  be  defined. 


COURSE  TITLE 


OBJECTIVE  f 


OBJECTIVE  ADEQUACY 

OBJECTIVE  CORRECTLY  STATED? _ 

REVISIONS 

CONDITIONS  y  n 

STANDARDS  y  n 

ACTION  y  n 


OBJECTIVE  CLASSIFIABLE?  (iN  ONE  AND  ONLY  ONE  WAY)  y 

(circle  the  classification)  REVISIONS 


REMEMBER 

USE-UNAIDED 

USE-AIDED 


FACT 

CONCEPT 

PROCEDURE 

RULE 

PRINCIPLE 


n 


OBJECTIVE  APPROPRIATE?  (FOR  JOB 

OR 

LATER 

training) 

CONDITIONS  APPROPRIATE? 

y 

n 

REVISIONS 

STANDARDS  APPROPRIATE? 

y 

n 

ACTION  -  TASK  LEVEL  APPROPRIATE? 

y 

n 

-  CONTENT  TYPE  APPROPRIATE? 

y 

n 

if  REMEMBER,  is  there  a  later 

USE  objective? 

y 

n 

if  USE-UNAIDED,  is  there  a  previous 
REMEMBER  objective? 

y 

n 

if  USE-AIDED,  is  the  aid  adequate 
or  are  other  objectives  on  the 
aid  included? 


y  rt 


COURSE  TITLE 


OBJECTIVE  Jf 


TEST  CONSISTENCY  and  ADEQUACY 

test  test  test 

item  item  item 


CONSISTENCY  # _  revisions  # _  revisions  H _  revisions 


CONDITIONS  SAME  (as  objective)? 

y  n 

y  n 

y  n 

STANDARDS  SAME? 

y  n 

y  n 

y  n 

ACTION  (TASK/CONTENT)  SAME? 

y  « 

y  n 

y  « 

TYPICAL  OF  JOB? 

y  n 

y  n 

y  n 

FORMAT  APPROPRIATE? 

y  n 

y  n 

y  n 

ADEQUACY 

1 

CLEAR? 

y  n 

y  n 

y  n 

UNAMBIGUOUS? 

y  n 

y  ri 

y  n 

WELL-CONSTRUCTED? 

y  n 

y  n 

y  n 

FREE  Of  HINTS? 

y  n 

y  n 

y  n 

REVISIONS 


Are  COMMON  ERRORS  tested?  y  n 

Are  there  ENOUGH  ITEMS?  y  n 


2 


JB1— ‘mrgm 


COURSE  TITLE 


OBJECTIVE  # 


PRESENTATION  CONSISTENCY  and  ADEQUACY  1 

CONSISTENCY 

REVISIONS 

Required  PRESENTATION 

y 

n 

COMPONENTS  Present? 

Required  PRESENTATION 

n 

COMPONENTS  Complete? 

y 

ADEQUACY 

STATEMENT  ADEQUATE? 

Separate? 

y 

n 

Identified? 

y 

n 

Clear? 

y 

n 

Hel  p? 

y 

n 

EXAMPLES  ADEQUATE? 

| 

Separate? 

y 

n  If 

Identi fied? 

y 

n  1 

Clear? 

y 

n 

Help? 

y 

n  ’ 

Matching  (concepts  only)? 

y 

n 

Easy  to  hard? 

y 

n 

Enough? 

y 

n  i 

Common  errors? 

y 

n 

PRACTICE  ADEQUATE? 

Separate? 

y 

n 

Identified? 

y 

n 

Clear? 

y 

n 

Free  of  hints? 

y 

n 

Easy  to  hard? 

y 

n 

Common  errors? 

y 

n 

Enough? 

y 

n 

Feedback  Separate? 

y 

n 

Feedback  Identified? 

y 

n 

Feedback  Help? 

y 

H  3 
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